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TO ALL WHOM IT MAY CONCERN: 

JBE IT KNOWN that We, Dz Ching Ju of Sunnyvale, California and citizen of the Taiwan, Kalyan 
IMuthukumar of Cupertino, California and citizen of India, Shankar Ramaswamy of Bethel Park, 
1 Pennsylvania, and citizen of India, and Barbara Bluestein Simons of Palo Alto, California, and citizen 
Z_ of the United States, have invented new and useful improvements in 



A METHOD OF, SYSTEM FOR, AND COMPUTER PROGRAM PRODUCT FOR 
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THROUGH CODE RESTRUCTURING 
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BACKGROUND OF THE INVENTION 

Us cx/y 

1. Field ofthe Invention 

This invention relates generally to optimizing compilers for development of computer 
programs for use on a computer, and more particularly to code restructuring. 

2. Description of the Related Art 

Advances in processor designs have provided very high speed instruction processing. In order 
to sustain this speed, processors need very effective usage of the memory hierarchy ofthe machines. 
Ensuring effective usage is difficult for large programs like compilers or database engine codes that 
have complex control flow. This occurs, for example, as a result of having to consider inputs of 
different types or handle error conditions appropriately. Consequently, code generated for large 
complex programs consists of sections that are heavily executed intermingled with sections that are 
infrequently executed, resulting in poor utilization of the memory hierarchy. Code restructuring may 
improve memory hierarchy utilization and, consequently, performance. 

The state ofthe art of code restructuring techniques is described in Simons (B. Simons, "Code 
Positioning for Procedures and Basic Blocks," IBM Technical Report ADTI- 1994-006; also available 
as IBM Santa Teresa Laboratory Technical Report TR 03.580.). Two representative techniques 
described by Simons include the IBM Heatshrink and the IBM AOPT products. Heatshrink, a trace- 
directed program restructuring (TDPR) tool, was designed and implemented by Randy R. Heisch. A 
more detailed description of Heatshrink is given in Heisch (R. R. Heisch, "Trace-Directed Program 
Restructuring for AIX Executables", IBM Journal of Research and Development, vol. 38, September 
1994.). AOPT, a stand-alone tool developed by Itai Nahshon, attempts to locate a "hot path" in the 
code and then move the entire path as a single entity, rather than move individual basic blocks. 
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1 Profile information, used by both Heatshrink and AOPT, is obtained by AOPT. Profiling involves 

2 running code using sample or representative inputs. Based on the behavior of the code on the given 

3 inputs, one makes assumption about its behavior on all inputs. Heatshrink and AOPT both perform 

4 global reordering of basic blocks and are bundled together as part of FDPR (Feedback Directed 

5 Program Restructuring tool), an IBM Licensed Program Product for IBM AIX Operating System, 

6 Version 4.1. 
7 

8 Pettis et al. (K. Pettis and R. C. Hansen, "Profile Guided Code Positioning," Proceedings of 

9 %he A CM SIGPIAN '90 Conference on Programming language Design and Implementation (PLDI), 

1 0 CH pp. 1 6-27, June 1 990.) follow a different approach, that also relies on profiling information, but that 

1 1 K moves executed basic blocks only within a procedure in addition to moving procedures. Basic blocks 

1 2 S that are not executed according to the profile data are moved to a separate region at the end of an 

13 ^ordering. 

14 ^ 

1 5 H These and other conventional techniques for code restructuring attempt to keep frequently 

1 6 g executed code close together in memory. Intuitively, such an approach tries to retain the working set 

1 7 r:5 of frequently executed code in cache. If the working set fits in the cache, this approach is reasonable; 

1 8 on the other hand, if the cache is small or if the effective portion of the cache available to an 

1 9 application is small due to multiprogramming, conventional code restructuring techniques may not 

20 perform well. A major flaw of these conventional approaches is that "closeness" in memory does not 

21 actually relate to cache or paging performance. Two basic blocks may be "close" together in 

22 memory, and yet there may still be a cache miss or page fault transitioning between the two basic 

23 blocks, or they may be very far apart and still cause the same effect of a cache miss or page fault. 

24 Thus, conventional code restructuring techniques fail to realize that blocks interacting frequently 

25 should be kept in the same cache line or page rather than "close" to each other in memory. 

26 Conventional code restructuring techniques also fail to realize that memory hierarchy hardware 

27 parameters such as cache line size, page size, cache associativity size and translation lookaside buffer 

28 (TLB) size may be considered in determining the code structure of an application. 
29 
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Thus, the conventional code restructuring techniques fail to consider memory hierarchy. As 
such, there is a need for a method of, system for, and computer program product for, providing code 
restructuring dependent upon a memory hierarchy. 
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I SUMMARY OF THE INVENTION 

2 

3 The invention disclosed herein comprises a method of, a system for, and an article of 

4 manufacture for providing code restructuring based on profiling information and memory hierarchy, 

5 that is also scalable to arbitrary levels of memory hierarchy. 
6 

7 The present invention reorders code to reduce communications across cache line and across 

8 page boundaries. The present invention also reorders code to reduce the likelihood that chunks of 

9 CScode that communicate with each other and that have the same cache or TLB associativity set will be 
1 0 S assigned to different cache lines or pages. The present invention may be generalized to deal with 

II multiple levels of cache or other types of memory hierarchy. 

12 £ 

1 3 il The present invention accomplishes this by reordering code such that chunks of code that 

14 j\ communicate extensively with each other are on the same cache line, page, or other level of memory 

1 5 p hierarchy. Reordering is performed by first clustering the nodes, performing first level padding of 

16 Q unfilled clusters with basic blocks that were not accessed in profiling, and then performing additional 

1 7 H padding using no-operation instructions (NOP or NO-OP) to fill out any unfilled portion of that level 

1 8 of the memory hierarchy. For example, to reorder code for a first level cache line, nodes are clustered 

1 9 to reduce communications across cache lines, any of these clusters which are unfilled clusters are 

20 padded with basic blocks that were not accessed in profiling, and then any unfilled cache lines are 

21 further padded with NOPs to fill out the cache lines. _ 
22 

23 The present invention avoids set collisions by considering both memory hierarchy size and 

24 associativity. Associativity means that there is a connection between a memory location and the place 

25 in the memory hierarchy to which the data for that memory location is assigned. 4-way associativity, 

26 for example, means that the cache is divided into four sets, and each memory location is uniquely 

27 mapped into one of those four sets. This mapping may be done by using two bits of the memory 

28 location address. If a cache line is 1 28 bytes in size, and if a memory word contains 4 bytes, then 32 

29 words will fit on one cache line. A primary objective of the associativity mapping is to have each 
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1 consecutive chunk of 32 words map to different associativity sets. If bits 6 and 7 of the memory 

2 location address are used to determine to which associativity set a 32 bit chunk maps, then adjacent 

3 32 word chunks map to different sets. A set collision occurs when all the cache lines in a particular 

4 set are full. This may occur even if one of the other sets has space. As conventional processor 

5 implementations hard wire the associativity between memory and cache sets, data may not arbitrarily 

6 occupy a cache line in an associativity set that differs from the one to which the particular memory is 

7 assigned. The present invention considers both memory hierarchy size and associativity to avoid such 

8 associativity set collisions by dividing the cache size by its associativity, i.e., if an 8K cache is 4-way 

9 Q set associative, then n2 = 2048 bytes. This n2 falls between the cache line size, nl, of 128 bytes and 

10 S the page size, n3, of 4096 bytes. Similarly, the TLB size divided by its associativity is considered. 

1 1 Alternative embodiments of the present invention may be modified to ignore various levels of the 

1 2 =P memory hierarchy; for example, an alternative embodiment may ignore associativity. 

13 2 

14 ;\ The present invention restructures code by constructing a Program Execution Graph (PEG) 

15 C3 corresponding to a level of the memory hierarchy, partitioning this PEG to reduce estimated memory 

1 6 q overhead costs, and constructing a PEG for a next level of the memory hierarchy from the partitioned 

1 7 32 PEG. The present invention may be performed by: 
18 

1 9 a. constructing a Program Execution Graph (PEG) from control flow and frequency information 

20 from a profile of the program to be restructured. The profile of the program is produced by 

21 running the program using sample or representative inputs. The present invention assumes 

22 that the profiled program behavior is representative of the general program behavior, i.e., that 

23 basic blocks that communicate in the profiling will communicate in general. This type of 

24 profiling assumption is commonly used in practice by those skilled in the art. The PEG is a 

25 weighted undirected graph comprising nodes representing basic blocks and edges representing 

26 transfer of control between pairs of basic blocks. The weight of a node is the number of bytes 

27 in the basic block it represents, and the weight of an edge is the frequency of transition in the 

28 profiling between the pair of basic blocks it connects. Those skilled in the art recognize that 

29 the unit of measurement for the weight may be a unit other than bytes, i. e., bits, words, etc. 
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1 b. partitioning or clustering the nodes of the PEG into clusters such that the sum of the weights 

2 of the nodes in any cluster is no greater than an upper bound, and the sum of the weights of 

3 the edges between the clusters is minimized. 
4 

5 c. constructing a next PEG from the clusters of the partitioned PEG such that a node in the next 

6 PEG corresponds to a cluster in the partitioned PEG, and such that there is an edge between 

7 two nodes in the next PEG if there is an edge between the clusters represented by the two 

8 nodes. 

9 Q 

10 old. assigning weights to the nodes and edges of the next PEG to produce a PEG. 

11 m 

12 =P e. repeating steps b through e for each level of the memory hierarchy. 

13 U 

14 L f. restructuring the basic blocks corresponding to the clusters into contiguous code starting at an 

1 5 p appropriate memory boundary, and adding NOPs as needed so that the number of memory 

16 C3 locations occupied by each cluster is equal to an upper bound of the cluster weight. 

17 g 

1 8 In accordance with one aspect of this invention, a Program Execution Graph constructor 

1 9 constructs a Program Execution Graph (PEG), a weighted undirected graph comprising nodes 

20 representing basic blocks or clusters of basic blocks and edges representing transfer of control 

21 between pairs of basic blocks where each basic block in the pair is in a different cluster, wherein a 

22 weight of a node of the PEG is a size of a represented basic block, and a weight of an edge is a 

23 frequency of transition between a pair of basic blocks that the edge connects or a frequency of 

24 transition between elements of two clusters that the edge connects. 
25 

26 In accordance with another aspect of this invention, the PEG, a weighted undirected graph G 

27 = (V,E) with edge and node weights, is partitioned into clusters such that the sum of the weights of 

28 , the edges whose endpoints are in different partitions is minimized and the sum of weights of nodes in 

29 each set is no greater than a given upper bound. 
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1 In accordance with another aspect of this invention, a PEG for level /' of a memory hierarchy, 

2 G, - (V„ E), is partitioned into clusters, and these clusters from level /" are the nodes for G, +/ = (V i+I , 

3 E HJ ), a PEG for level /+/ of the memory hierarchy. 
4 

5 In accordance with another aspect of this invention, if u and v e V^,, there is edge (u,y) 

6 between u and v if and only if there is an edge between some pair of nodes in G h one of which is in 

7 the cluster corresponding to u and one of which is in cluster corresponding to v. 
8 

9 In accordance with another aspect of this invention, the weight of edge (w,v) in G, +/ is the sum 

10 of the weights of the edges between component nodes of u and v in G,. 

11 5 

1 2 5j In accordance with another aspect of this invention, the upper bound for a level of the 

1 3 ^ memory hierarchy may be equal to the size of that level of the memory hierarchy. 

14 H 

15 H In accordance with another aspect of this invention, the upper bound for a level of the 

1 6 memory hierarchy may be equal to a multiple of the size of that level of the memory hierarchy. 

17 6 

1 8 The present invention has the advantage of restructuring code responsive to a memory 

19 hierarchy. 
20 

21 The present invention has the further advantage of reducing memory hierarchy access cost of 

22 restructured code. 
23 

24 The present invention has the further advantage of reducing cache misses between basic 

25 blocks of restructured code. 
26 

27 The present invention has the further advantage of reducing page faults between basic blocks 

28 of restructured code. 
29 
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1 The present invention has the further advantage of restructuring code responsive to multiple 

2 levels of cache or other types of memory hierarchy. 
3 

4 The present invention has the further advantage of being extendable to deal with additional 

5 memory hierarchies, such as second level (L2) or higher caches that may be introduced in future 

6 machines. 
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1 BRIEF DESCRIPTION OF THE DRAWINGS 

2 

3 For a more complete understanding of the present invention and the advantages thereof, 

4 reference is now made to the Detailed Description in conjunction with the attached Drawings, in 

5 which: 
6 

7 Figure 1 shows a Program Execution Graph G, of a first memory hierarchy level; 

8 

9 y Figure 2 shows a partitioned Program Execution Graph G, produced in accordance with the 

1 0 ff 1 present invention; 

11 [i 

1 2 j Figure 3 shows a Program Execution Graph G 3 of a second memory hierarchy level produced 

1 3 » u from the clusters of the partitioned Program Execution Graph G, in accordance with the present 

14 j=+ invention; 

15 £ 

16 2 Figure 4 shows a partitioned Program Execution Graph G 3 produced in accordance with the 

17 C3 present invention; 
18 

19 Figure 5 shows a Program Execution Graph G 3 of a third memory hierarchy level produced 

20 from the clusters of the partitioned Program Execution Graph G 2 in accordance with the present 

21 invention; — 
22 

23 Figure 6 and Figure 7 are flowcharts illustrating the operations preferred in carrying out the 

24 present invention; and 
25 

26 Figure 8 shows a computer system used in performing the method of the present invention, 

27 forming part of the apparatus of the present invention, and which may use the article of manufacture 

28 comprising a computer-readable storage medium having a computer program embodied in said 

29 medium which causes the computer system to practice the present invention. 
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1 DESCRIPTION OF THE PREFERRED EMBODIMENTS 

2 

3 ' 

4 Referring now to Figure 1 through Figure 5, sample Program Execution Graphs are shown 

5 to illustrate the practice of the present invention. The control flow and frequency information of the 

6 profiled program is organized in the form of a Program Execution Graph (PEG) which is a weighted 

7 undirected graph. For example, Figure 1 illustrates a PEG G„ generally referred to as 101, where G, 

8 = ( V„ E,). A node in the set of nodes V, corresponds to a basic block that has been executed at least 

9 r g once during profiling. For G, 101, the set of nodes V, comprises the set {A 103, B 105, C 107, D 

10 j| 109, E 111, F 113, G 115, H 117, 1 119, J 121, K 123, L 125, M 127, N 129, 0 131, P 133, and Q 

11 W 135} . An edge (u, v), in the set of edges E„ between nodes u and v implies that there is some 

1 2 =E transition that occurs during profiling between the two basic blocks represented by u and v. For G, 

13 ii 100, the set of nodes E, comprises the set {(A, B) 137, (B, C) 139, (C, F) 141, (F, K) 143, (K, L) 

14 ^ 145, (L, N) 147, (N, O) 149, (O, P) 151, (O, Q) 153, (B, N) 155, (F, J) 157, (J, K) 159, (B, E) 161, 

15 G (E, F) 163, (B, D) 165, (D, E) 167, (E, G) 169, (G, I) 171, (I, L) 173, (G, H) 175, (H, I) 177, (H, M) 

16 □ 179, (M, N) 181, and (D, M) 183}. The weight of a node v, written W,(v), is the size (in bytes) of 

17 J= the basic block that v represents. In G, 101, node A 103 has a weight W,(A) of 1 14 bytes, node B 

1 8 105 has a weight W,(B) of 36 bytes, C 107 has a weight W,(Q of 8 bytes, and so on. The weight of 

19 edge e = (u,v), written W,(e), is the total number of transitions from basic block u to basic block v 

20 plus the total number of transitions from basic block v to basic block u. In G, 101, edge (A, B) 137 

21 has' a weight W,((A, B)) of 556 transitions, edge (B,C) 139 has a weight W,((B,C)) of 745 transitions, 

22 edge (C, F) 141 has a weight W,((C, F)) of 745 transitions, and so on. 
23 

24 Using the initial PEG G, for a program and given all the parameters for a n-level memory 

25 hierarchy, the present invention performs code restructuring in n or fewer steps. The i' h step 

26 constructs a new PEG by clustering PEG G, wherein graph partitioning is performed on G, to 

27 produce the clustering. Clustering and graph partitioning are discussed in the following paragraphs. 
28 

29 
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1 A cluster , written C7, is defined to be a set of nodes. A clustering of a graph G with upper 

2 bound M written Chis((G,N), is a partitioning of the nodes in G into clusters so that the sum of the 

3 node weights of the nodes in any cluster is no greater than N. Each cluster in a partitioned PEG 

4 represents a unit in the level of the hierarchy for which clustering is performed. For example, in the 

5 first stage, a unit represents the size of a cache line (in bytes) or some integral multiple of this size (to 

6 allow for basic blocks that are larger than a cache line). 
7 

8 The goal of a graph partitioning heuristic is to create clusters of nodes from the PEG such that 

9 rathe sum of the weights of the edges that have endpoints in different clusters is minimized, subject to 

1 0 Sthe total weight of the nodes in each cluster not exceeding N. This goal is modeled by using a cost 

1 1 f| function CO. For edge e, e has zero cost in Clust(G.N), written C(e) = 0, if both endpoints of e are in 

1 2 Ip the same cluster for Clust(G.N) . Otherwise, e has a cost in Clusl(G,N), C(e) = 1 . Those skilled in the 

13 2 art recognize that costs other than zero or one may be assigned by the cost function for the purposes 

14 ; of partitioning the graph. The goal is to compute a clustering that minimizes the sum of C(e) times 

15 Q W(e) for all edges e in E. The result of graph partitioning is to create the clusters, i.e., to assign 

16 n values to C(e) for all edges e in E. lfC(e) = 0, then the endpoints of edge e are in the same cluster; 

17 otherwise, the endpoints of edge e are in different clusters. 
18 

19 Each cluster represents a unit in the stage of the hierarchy for which clustering is performed. 

20 Because a basic block may be larger than a cache line, the first stage of the heuristic allows a cluster 

21 to be the size of some multiple of the number of bytes in a cache line. At each subsequent stage, the 

22 clusters are limited to the size of the portion of the memory hierarchy being considered, for example, 

23 a memory page size. However, if for some reason this cluster size is not preferred, the heuristic may 

24 be easily modified to assign a different value to the cluster size. 
25 

26 The cost function is an approximation of the memory hierarchy overhead incurred by the 

27 restructured program on a real machine. Formally, the code restructuring problem may be defined as 

28 a first minimization of & h then a minimization of then a minimization of 0 3 , ... , and a final 

29 minimization of <P h where k is the number levels in the memory hierarchy, and where: 
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1 <?, = X c ^ xW T^ 

2 eel-, 
3 

4 The objective is to construct a clustering Clust(G,NJ so that <P, is the summation of the product of 

5 C(e) and W(e) for all e in £,; where C(e) = 0 if both endpoints of e are in the same cluster and C(e) = 

6 1 otherwise; where the sum of the node weights of the nodes in each cluster in ClustfG^N) is no 

7 greater than N h and where N, is some multiple of the cache line size. 
8 

9 f =* For example, if a cache line size is 1 28 bytes, N, may be set to 1 28 bytes, and G, may be 

1 0 ~ partitioned into clusters such that the sum of the node weights of the nodes in each cluster is no 

1 1 CO greater than 1 28 bytes where N, is a multiple of one times the cache line size. Such a partitioning of 

12 S G, yields a clustering Clus«G „N ',) = {{A 103}, (B 105, C 107, F 113}, {J 121}, {K 123, L 125}, {D 

13 | 109, E 111}, {G 115, H 117, 1 119}, {M 127, N 129}, {O 131, P 133, Q 135}}. Clustering 

14 3 Clust(G„N]) is illustrated in Figure 2 where cluster 205 comprising {A 103} has a node weight sum 

15 5 of 1 14 bytes; cluster 210 comprising {B 105, C 107, F 113} has a node weight sum of 126 bytes; 

16 K cluster 215 comprising {J 121} has a node weight sum of 92 bytes; cluster 220 comprising {K 123, L 

17 1 125} has a node weight sum of94 bytes; cluster 225 comprising {D 109, E 111} has a node weight 

18 " sum of 86 bytes; cluster 230 comprising {G 115, H 117, 1 119} has a node weight sum of 120 bytes; 

19 cluster 235 comprising {M 127, N 129} has a node weight sum of 80 bytes; and cluster 240 

20 comprising {O 131, P 133, Q 135 } has a node weight sum of 104 bytes. 

21 - 

22 Once a clustering on G, is constructed, "padding" is performed by adding basic blocks that 

23 were not executed during the profiling to clusters that have capacity, i.e. where the sum of the 

24 weights of the nodes in the cluster is less than N, and where adding the un-executed basic block will 

25 not cause the sum of the cluster's weights to exceed N,. If there are still basic blocks remaining that 

26 have not been added to any of the clusters, and if there is no more cluster capacity to absorb these 

27 basic blocks, then they are grouped into additional clusters of their own. The technique of adding th 

28 un-executed basic block nodes to the clusters or of putting them into their own clusters is called bin 

29 packing. A greedy bin packing heuristic may be used for performing the bin backing. 
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1 Although the example of Figure 2 does not contain any un-executed basic blocks that may be 

2 used for padding (as the initial PEG does not include un-executed basic blocks in its node set), if 

3 there is an additional unexecuted basic block whose size is less than 48 bytes, it may be padded into 

4 the cluster 235 with node M 127 and node N 129 yielding a padded cluster whose sum of node 

5 weights does not exceed N, = 1 28. In a similar manner, a greedy bin packing heuristic may pad the 

6 other clusters. 
7 

8 Once all of the basic blocks have been put into some cluster, then the next stage is performed 

9 O which is the construction of a graph G 2 = (V 2 ,E 2 ). Each node in V 2 corresponds to a cluster in G„ 

1 0 S and each node in V 2 has a weight of one. This weight corresponds to the assignment of N, bytes to 

1 1 ?f the basic blocks in each cluster. An edge e = (u,v) exists in E 2 if and only if both u and v are in V 2 , u 

12 =p and v correspond to different clusters, and there is some edge in E, from one of the nodes in the 

1 3 Si cluster represented by u to some node in the cluster represented by v, or visa versa. The weight of e 

14 r is the sum of the weights of all edges in E, that have endpoints in the two clusters (u and v). 

15 Q 

16 S Referring next to Figure 3, a graph G 2 = (T* £^ constructed from the clusters C/i^/CG/,^ 

1 7 j= of G, is illustrated. G 2 is constructed by creating a node corresponding to each cluster of 

1 8 Clt4sl(G,.N,) such that node A 305 corresponds to cluster 205, node BCF 310 corresponds to cluster 

19 210, node J 315 corresponds to cluster 215, node KL 320 corresponds to cluster 220, node DE 325 

20 corresponds to cluster 225, node GHI 330 corresponds to cluster 230, node MN 335 corresponds to 

21 cluster 235, and node OPQ 340 corresponds to cluster 240. Each of the nodes V 2 = {A 305rBCF 

22 310, J 315, KL 320, DE 325, GHI 330, MN 335, OPQ 340 } is assigned a weight of one. Edges E 2 

23 are then constructed between various nodes V 2 where an edge e = (u, v) exists in E 2 if and only if both 

24 u and v are in V 2 , u and v correspond to different clusters, and there is some edge in E, from one of 

25 the nodes in the cluster represented by u to some node in the cluster represented by v, or visa versa. 

26 For example, edge (BCF, DE) 350 is created as there is an edge (B, D) 165 between node B 105 and 

27 node D 109, and as node B 105 and node D 109 are in different clusters, node B 105 being in cluster 

28 210 and node D 109 being in cluster 225. In a similar manner, edges (A, BCF) 345, (BCF, KL) 355, 

29 (BCF, J) 365, (BCF, MN) 370, (DE, MN) 375, (DE, GHI) 380, (GHI, MN) 385, (GHI, KL) 390, (J, 
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1 KL) 392, (KL, MN) 394, and (MN, OPQ) 396 are created. Weights are then assigned to the edges of 

2 E-, where the weight of an edge (m, v) in E 2 is the sum of the weights of all edges in E, that have 

3 endpoints in the two clusters w and v. For example, edge (BCF, DE) 350 is assigned a weight of 

4 440, the sum of the edge weights of edge (B, D) 165, edge (B, E) 161, and edge (E, F) 163, all of 

5 these edges having endpoints in cluster 210 and cluster 225. In a similar manner, edge weights of 

6 556, 700, 750, 88, 200, 360, 800, 112, 128, 470, and 500, respectively, are assigned to edges (A, 

7 BCF) 345, (BCF, KL) 355, (BCF, J) 365, (BCF, MN) 370, (DE, MN) 375, (DE, GFfl) 380, (GUI, 

8 MN) 385, (GHI, KL) 390, (J, KL) 392, (KL, MN) 394, and (MN, OPQ) 396. 

9 s 

1 0 CH Let w, be the size of the next level of the memory hierarchy, typically the page size in bytes. 

11 Iff Set N, = n/N,. Then the same clustering heuristic is applied to G,, using the same cost function C, 

1 2 m i.e. counting only those edges that have end points in different clusters in G 2 . This partitioning of G, 

1 3 H is used to construct graph G 3 = (N* EJ by making each node in V 3 correspond to a cluster in V* and 

14 N= having an edge in E 3 if and only if there is some edge xnE 2 that has end points in nodes in the two 

15 2 clusters represented by the endpoints ofthe£, edge. Again the nodes have a weight of one, and the 

1 6 S J edges have weights corresponding to the sum of the weights on the edges in G 3 that run between 

17 C3 some node in cluster u and some node in cluster v. 
18 

! 9 For example, if w,, the size of the next level of the memory hierarchy, is 5 1 2 bytes, 

20 then set N, = n/N, = 5 12/128 = 4. G, may be partitioned into clusters such that the sum of the node 

21 weights of the nodes in each cluster is no greater than 4. Such a partitioning of G, yields a clustering 

22 Clus1(G*NJ = { ABCFJKL 410 and DEGHIMNOPQ 420) as illustrated in Figure 4. A graph G 3 = 

23 (V 3 , EJ may be constructed from the clusters OustfGyNJ of G 2 as "illustrated in Figure 5. G 3 is 

24 constructed by creating a node corresponding to each cluster of ClusltG^NJ such that node 

25 ABCFJKL 510 corresponds to cluster 410, and node DEGHIMNOPQ 520 corresponds to cluster 

26 420. Each of the nodes V 3 = { ABCFJKL 510, DEGHIMNOPQ 520 } is assigned a weight of one. 

27 Edges E 3 are then constructed between various nodes V 3 where an edge e = (u, v) exists in E 3 if and 

28 only if both u and v are in V 3 , u and v correspond to different clusters, and there is some edge in E 2 

29 from one of the nodes in the cluster represented by u to some node in the cluster represented by v, o 



ST9-95-010 



16 



# 



1 visa versa. For example, edge (ABCFJKL, DEGHIMNOPQ) 530 is created as there is an edge (BCF, 

2 DE) 350 between node BCF 310 and node DE 325 which are in different clusters, node BCF 310 

3 being in cluster 410 and node DE 325 being in cluster 420. Weights are then assigned to the edges 

4 of E 3 where the weight of an edge (m, v) in E 3 is the sum of the weights of all edges in E 2 that have 

5 endpoints in the two clusters u and v. For example, edge (ABCFJKL, DEGHIMNOPQ) 530 is 

6 assigned a weight of 1 1 1 0, the sum of the edge weights of edge (BCF, DE) 350, edge (KL, GHI) 

7 390, edge (KL, MN) 470, and edge (BCF, MN) 370, all of these edges having endpoints in cluster 

8 510 and cluster 520. 

9 o 

1 0 1-2 This process of graph partitioning may be repeated for each level of the memory hierarchy for 

11 W /' > /. For level /' of the memory hierarchy, let w, be the size of the memory hierarchy, and set N, = 

12 Ic n/Ni.,. Then the same clustering heuristic is applied to G h using the same cost function C, i.e. 

1 3 ^ counting only those edges that have end points in different clusters in G t . This partitioning of G, is 

14 2 . used to construct graph G, +/ = (N i+I , E,J by making each node in V i+] correspond to a cluster in V„ 

1 5 b and having an edge in E M if and only if there is some edge in E, that has end points in nodes in the 

16 S two clusters represented by the endpoints ofthe£, +/ edge. Again the nodes have a weight of one, 

17 § and the edges have weights corresponding to the sum of the weights on the edges in G, that run 

18 between some node in cluster u and some node in cluster v. 
19 

20 One of the advantages of the present invention is that it may be generalized to deal with 

2 1 memory hierarchies with an arbitrary number of levels. In particular, the present invention-may be 

22 applied to future improvements to memory hierarchy hardware. 
23 

24 Another advantage of the present invention is that it may be modified to deal with special 

25 cases. An example of such a special case occurs when a pair of basic blocks need to be close in 

26 memory. In this case, the edge connecting the two basic blocks may be given an exceedingly large 

27 weight, thereby forcing the two basic blocks into the same cluster if together they don't exceed the 

28 upper bound on the cluster size. This approach may also be generalized to the case in which several 

29 basic blocks are required to be close together in memory. 
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1 The present invention may also be modified to deal with the case of a program with a few very 

2 large basic blocks. Assume basic blocks B h Bj, and B k have weights larger than the size of the cache 

3 line, and that the weights of all the other basic blocks are no greater than the cache line size. Define 

4 Q> t = g, - {B„ Bj, BJ, i.e. the PEG that is obtained by removing B h B p and B k from the problem input. 

5 Let N, - the cache line size, and let Clust(G'„ N,) be the clustering that is computed by the heuristic. 

6 Clusl(G'„ NJ + {B„ Bj, BJ may then be used as the input node set to the next stage of the heuristic. 
7 

8 As the nodes representing B h B p and B k have weights greater than the value used for N, in the 

9 O first stage of the partitioning, the weight of the node in V 3 that corresponds to B s \sp h where/;, is the 

1 0 J smallest integer such that p t * N, >= the number of bytes in B t . Values Pj and p k for B J and B h 

1 1 respectively, are similarly computed. The heuristic may then be applied to G.„ with the above 

1 2 ± modifications. Note that this assumes that none of B h B p and B k is larger in bytes than n 2 . If that is 

1 3 U not the case, then again the very large basic block(s) may be removed from consideration and 

1 4 L, reintegrated at the next stage of the heuristic. 

15 5 

1 6 H Referring next to Figure 6 and Figure 7, flowcharts illustrating operations preferred in 

17 3 carrying out the present invention are shown. In the flowcharts, the graphical conventions of a 

1 8 diamond for a test or decision and a rectangle for a process or function are used. These conventions 

1 9 are well understood by those skilled in the art, and the flowcharts are sufficient to enable one of 

20 ordinary skill in the art to write code in any suitable computer programming language. 

21 - 

22 Referring now to Figure 6, the process of the invention, generally referred to as 600, begins 

23 at process block 610. Thereafter, process block 615 determines memory hierarchy parameters such 

24 as the number of levels in the memory hierarchy and the size of each such level, and process block 

25 620 profiles the target program. The profiler gathers information comprising identification of basic 

26 blocks, control flow between basic blocks, and frequencies of transition between basic blocks. The 

27 present invention assumes a program is divided into basic blocks, where a basic block is a contiguous 

28 sequence of instructions with a single entry into and a single exit out of this sequence. In addition, 

29 the present invention assumes information about the transfer of control between basic blocks and the 
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1 frequency with which the transfer occurs during actual program execution. This information may be 

2 made available through the use of profiling techniques well known to those skilled in the art such as 

3 the teachings summarized in Simons (B. Simons, "Code Positioning for Procedures and Basic 

4 Blocks," IBM Technical Report ADTI- 1994-006; also available as IBM Santa Teresa Laboratory 

5 Technical Report TR 03.580.). Responsive to the profiler provided information, process block 625, a 

6 Program Execution Graph constructor, constructs a Program Execution Graph (PEG), a weighted 

7 undirected graph comprising nodes representing basic blocks and edges representing transfer of 

8 control between pairs of basic blocks. In this PEG produced by the Program Execution Graph 

9 5 = constructor, a weight of a node of the PEG is a size of a represented basic block, and a weight of an 

1 0 "0 edge is a frequency of transition in the profiling between a pair of basic blocks that the edge connects. 

1 1 W Process block 630 then initializes a memory hierarchy level / to one, and sets n, equal to a smallest 

1 2 % power of 2 that is a multiple of the cache line size and that is greater than the sizes of all basic block 

1 3 ® sizes. Thereafter, process block 635 applies a heuristic H to partition G, into CIust(G„N,), a 

14 s partitioning of the nodes in G, into clusters so that the sum of the node weights of the nodes in any 

1 5 J i cluster is no greater than N,. The heuristic H solves the relaxed graph partitioning problem, and such 

16 ~ heuristics are well known to those skilled in art such as the teachings of Barnard et al. (S. T. Barnard 

1 7 O and H. D. Simon, "Fast Multilevel Implementation of Recursive Spectral Bisection for Partitioning 

18 . = Unstructured Problems," Concurrency: Practice mid Experience, vol.6(2), pp. 101-1 17, April 1994.). 

1 9 After clustering is performed on G„ process block 640 pads the clusters with unexecuted nodes by 

20 adding basic blocks that were not executed during the profiling to clusters that have capacity, i.e. 

21 where the sum of the weights of the nodes in the cluster is less than N, and where adding the. 

22 un-executed basic block will not cause the sum of the cluster's weights to exceed N,. This padding is 

23 performed to reduce unused memory. If there are still basic blocks remaining that have not been 

24 added to any of the clusters, and if there is no more cluster capacity to absorb these basic blocks, then 

25 process block 640 groups these unexecuted nodes into additional clusters of their own. The 

26 technique of adding the un-executed basic block nodes to the clusters or of putting them into their 

27 own clusters is called bin packing. A greedy bin packing heuristic, which is well known to those 

28 skilled in art such as the teachings of Garey et al. (Garey, Michael and Johnson, David S. Computers 

29 and Intractability, A Guide to the Theory of NP Completeness, pages 124-127, W. H. Freeman & 
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1 Co., San Francisco, Ca., 1979) may be used to perform the bin backing. Process block 645 then 

2 creates the next level PEG G 2 from the level 1 partitioned PEG Clusl(G„NJ. In this graph G, = (V* 

3 Ej) constructed by process block 645, each node in V 2 corresponds to a cluster in G,, and each node 

4 in V 2 has a weight of one. This weight corresponds to the assignment of N, bytes to the basic blocks 

5 in each cluster. An edge e = (u,v) exists in E 2 if and only if both u and v are in V 2 , u and v correspond 

6 to different clusters, and there is some edge in E, from one of the nodes in the cluster represented by 

7 // to some node in the cluster represented by v, or visa versa. The weight of e is the sum of the 

8 weights of all edges in E, that have endpoints in the two clusters (« and v). 

9 O 

1 0 In Higher levels of the memory hierarchy, / greater than 1 , are processed by a loop comprising 

1 1 m decision block 650 and process blocks 655, 660, 665, and 670. Decision block 650 begins the loop 

1 2 !:£ for each level / of the memory hierarchy. As the loop begins, decision block 650 determines if level / 

13 Hms less than or equal to a maximum number of levels. If so, then process block 655 increments level /', 

14 H= and then process block 660 sets n, equal to the size of the memory hierarchy for level i, i.e., to the 

1 5 i% page size in bytes for the page level of the memory hierarchy. Thereafter, process block 665 applies a 

16 O heuristic H to partition G, into clusters having sizes no greater than N t = w/A^, to produce 

1 7 C3 Clust(G i .n/N l _ l ). Process block 665 may perform this clustering in the same manner as process block 

1 8 635 or it may use a different partitioning heuristic. Thereafter, process block 670 creates the next 

19 level PEG G, +/ from the level /' partitioned PEG ClustfG^n/NJ. In this graph G (+ , = (V„„ E^ 

20 constructed by process block 670, each node in V^, corresponds to a cluster in G h and each node in 

21 V t ;, has a weight of one. An edge e = (u,v) exists in £, +/ if and only if both u and v are in V-^„ u and v 

22 correspond to different clusters, and there is some edge in £, from one of the nodes in the cluster 

23 represented by // to some node in the cluster represented by v, or visa versa. The weight of e is the 

24 sum of the weights of all edges in E, that have endpoints in the two clusters (w and v). Process block 

25 670 performs this PEG construction in the same manner as process block 645. Thereafter, processing 

26 loops back to decision block 650, the beginning of the loop, to process the next level /+/. 
27 

28 Returning now to decision block 650, if level / is not less than or equal to a maximum number 

29 of levels, then process block 675 reorders basic blocks in memory by grouping all of the nodes of a 
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1 cluster in an adjacent order; process block 680 adds NOPs to fill out the unused portion of memory 

2 for that memory hierarchy level for each level of the memory hierarchy; process block 685 stores the 

3 reordered and padded basic blocks in memory beginning With the first memory location after a 

4 boundary for all the memory levels; and then the process ends at process block 690. 
5 

6 Referring now to Figure 7, an expansion of process block 670 is illustrated. After process 

7 block 665 partitions the PEG, then process block 710, process block 720, process block 730, and 

8 process block 740 construct a Program Execution Graph G I+/ for the next higher level i + 1 wherein 

9 f =i the nodes N i+I of G <+/ are constructed from the clusters Clust(G i ,n/N i J of G,. Process block 720 

1 0 ;M assigns a weight of one to each node in N iA .,. Process block 730 assigns edges £ f+/ in G, +/ by placing 

11 CO an edge (m,v) between u and v, for each u and v of N i+I , if and only if there is an edge between a pair 

1 2 Ie of nodes in G„ one of which is in the cluster corresponding to u and one of which is in cluster 

1 3 rf corresponding to v. Process block 740 assigns weights to the edges in G l+/ ; the weight of an 

14 3 edge (u,v)inG /+/ is the sum of the weights of the edges between component nodes in G, of the 

1 5 f 3 clusterSf corresponding to u and v. Thereafter, processing loops back to decision block 650, the 

16 S beginning of the loop, to process the next level i+ 1. 

17 55 

18 In an alternative embodiment of the present invention, only a single loop may be used for 

19 processing all levels of the memory hierarchy. In this alternative embodiment, process block 665 may 

20 be modified to include the functionality of process block 635 so that process block 665 partitions all 

2 1 PEG's. Process block 670 may also be modified to include the functionality of process block 645 so 

22 that it creates the next level PEG from all partitioned PEG's. Process block 640 may be moved within 

23 the loop to pad the first or additional level clusters. 
24 

25 After the clustering is completed, the basic blocks are reordered in memory by grouping all of 

26 the nodes of a cluster in an adjacent order (process block 675), although the ordering of the basic 

27 blocks within this adjacent grouping may be varied. Because clusters must not cross boundaries of 

28 memory hierarchies, NOPs are added to fill out the portion of a memory hierarchy level that is not 

29 filled by the clusters. For memory hierarchy level 1, NOPs are added to the end of any clusters that 
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1 have size less than some multiple of the cache line size so that the amount of memory required for 

2 each cluster is some integral multiple of the cache line size. For each subsequent level of the memory 

3 hierarchy, the minimum number of NOPs necessary to fill out the unused portion of memory for that 

4 memory hierarchy level are added (process block 680). After the processing for the last memory 

5 hierarchy is completed yielding the final PEG, the reordered basic blocks are stored in memory 

6 starting at a boundary of all the memory levels so that all of the basic blocks from one cluster are 

7 adjacent, i.e. interleaving from different clusters is not allowed (process block 685). This is true for 

8 _ = every level of the hierarchy. 

9 5 

10 ti' Referring now to Figure 8, a block diagram illustrates a computer system 800 used in 

1 1 -0 performing the method of the present invention, forming part of the apparatus of the present 

1 2 CO invention, and which may use the article of manufacture comprising a computer-readable storage 

13 I = medium having a computer program embodied in said medium which may cause the computer system 

1 4 S to practice the present invention. The computer system 800 includes a processor 802, which includes 

15 S a central processing unit (CPU) 804, a memory 806, and a multi-level memory cache 810 (level 1 

16 3 cache (LI) 812 through level n cache (Ln) 814). Additional memory, in the form of a hard disk file 

1 7 M storage 816 and a computer-readable storage device 818, is connected to the processor 802. 

1 8 Computer-readable storage device 818 receives a computer-readable storage medium 820 having a 

1 9 computer program embodied in said medium which may cause the computer system to implement the 

20 present invention in the computer system 800. The computer system 800 includes user interface 

2 1 hardware, including a mouse 822 and a keyboard 824 for allowing user input to the processor 802 

22 and a display 826 for presenting visual data to the user. The computer system may also include a 

23 printer 828. 
24 

25 Although the present invention has been particularly shown and described with reference to a 

26 preferred embodiment, it will be understood by those skilled in the art that various changes in form 

27 and detail may be made without departing from the spirit and the scope of the invention. 
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